TMap supports the correct execution of the structured test process with a complete tool box. The tool box focuses on working with the following
subjects:

• Techniques: how it is tested
• Infrastructure: where and with what it is tested
• Organisation: who does the testing

The various tools are described in more detail in TMap at the moment they can be used. With the tool box, the tester possesses a great number of options to meet the test challenge successfully.

TECHNIQUES

Many techniques can be used in the test process. A test technique is a combination of actions to produce a test product in a universal manner. TMap provides techniques for the following (see figure 1):

• Test estimation
• Defect management
• Creating metrics
• Product risk analysis
• Test design
• Product evaluation.

Figure 1: Test techniques

TMap also offers various checklists and overviews that can be used as a tool during the preparation and/or execution of certain activities. The (groups of) test techniques are summarised in the following sections.

Test estimation

Estimates can be made at a number of different levels. The various estimation levels are shown in the figure below.


Figure 2: Estimation levels

Independent of the level, creating an estimate consists of the following generic steps:

1. Inventory the available material that can serve as a basis for the estimate.
2. Select (a number of) estimating techniques.
3. Determine the definitive estimate.
4. Present the outcome.

Choosing the estimating techniques in particular is a step requiring experience. You can select from several estimating techniques:

• Estimation based on ratios. Here, the test effort is generally measured against the development effort, e.g. in percentage ratios.
• Estimation based on test object size.
• Estimation using a ‘Work Breakdown Structure’.
• Proportionate estimation based on the total test budget.
• Estimation on the basis of extrapolating experience figures from the beginning of the testing programme.
• Estimation on the basis of size and strategy using TMap’s test point analysis (TPA).

Furthermore, TMap provides a technique to create an evaluation estimate.

Defect management

A defect is an observed difference between the expectation or prediction and the actual outcome. While the administration and monitoring of the defects is factually a project matter and not one of the testers, testers are usually very closely involved. A good administration must be able to monitor the lifecycle of a defect and provide various overviews. These overviews are used, among other things, to make well-founded quality statements.

Creating metrics

The definition, maintenance and use of metrics is important to the test process because it enables the test manager an answer, supported by facts, to questions like:

• What about the quality of the test object?
• What about the progress of the test process?

A structured approach to realise a set of test metrics is using the Goal-Question-Metric (GQM) method. In addition to describing the GQM method, TMap gives instructions to set up a practical test metrics starter set. It also provides a checklist that can be useful to make pronouncements on the quality of the object to be tested and the quality of the test process.

Product risk analysis

A product risk analysis (PRA) is analysing the product to be tested with the aim of achieving a shared view, among the test manager and other stakeholders, of the more or less risky characteristics and components of the product to be tested so that the thoroughness of testing can be agreed upon. The focus in PRA is on the product risks, i.e. what is the risk to the organisation if the product does not have the expected quality?

The result of the PRA constitutes the basis for the subsequent decisions in strategy as to light, thorough or non testing of a characteristic (e.g. a quality characteristic) or object part (component) of the product to be tested.

Test design

A test design technique is a standardised method to derive, from a specific test basis, test cases that realise a specific coverage. The implementation of test design techniques and their definition in the test specifications have several advantages:

• It provides a well-founded elaboration of the test strategy: the agreed coverage in the agreed place.
• It is a more effective way to detect defects than e.g. ad-hoc test cases.
• The tests are reproducible because the order and content of the test execution are described in detail.
• The standardised method ensures that the test process is independent of the individual who specifies and executes the test cases.
• The standardised method ensures that the test specifications are transferable and maintainable.
• It becomes easier to plan and manage the test process because the processes of test specification and execution can be split up into clearly definable blocks.

Test design techniques exist in many variants and combinations. The test design techniques described in TMap constitute a varied set with which most organisations can get to work immediately. TMap describes the following coverage types and test design techniques.

• Coverage types - paths, decision points, equivalence classes, pair-wise testing, orthogonal arrays, limit value analysis, CRUD, operational and load profiles, right and fault paths, and checklists
• Test design techniques - decision table test, data combination test, elementary comparison test, error guessing, exploratory testing, data cycle test, process cycle test, reallife test, semantic test, syntactic test, and use case test.

Product evaluation

TMap describes and uses the following evaluation techniques:

• Inspection - In addition to determining whether the solution is adequately processed, an inspection focuses primarily on achieving consensus on the quality of a product.
• Review - A review focuses primarily on finding courses for a solution on the basis of the knowledge and competencies of the reviewers, and on finding and correcting defects.
• Walkthrough - A walkthrough is a method by which the author explains the contents of a product during a meeting. Several objectives are possible:
  • Bringing all participants to the same starting point, e.g. in preparation for a review or inspection process
  • Transfer of information, e.g. to developers and testers to help them in their programming and test design work, respectively
  • Asking the participants for additional information
  • Letting the participants choose from the alternatives proposed by the author.

Various checklists and overviews

TMap offers a great variety of checklists that will constitute a welcome addition to the tester when executing certain activities. For instance, there are checklists that can be used as support in taking stock of the assignment, determining the test facilities, determining the test project risks, establishing the test strategy, the evaluation of the test process, taking interviews, and determining whether adequate information is available to use a specific test design technique. TMap also offers other tools, such as an overview matrix of automated tools per TMap activity, a test type overview, and criteria to select a tool.

These tools and many more can be found on and downloaded from www.tmap.net.

INFRASTRUCTURE

Test environments, test tools and workplaces are necessary to execute tests.

Test environments

A fitting test environment is necessary for dynamic testing of a test object (running software). A test environment is a system of components, such as hardware and software, interfaces, environmental data, management tools and processes, in which a test is executed. The degree to which it can be established in how far the test object complies with the requirements determines whether a test environment is successful. The setup and composition of a test environment therefore depend on the objective of the test. However, a series of generic requirements with which a test environment must comply to guarantee reliable test execution can be formulated. In addition to being representative, manageable and flexible, it must also guarantee the continuity of test execution.

Setting up and managing the test environment represents an expertise of which testers generally have no knowledge. This is why a separate department – outside the project – is generally responsible for setting up and managing the test environment.

Test tools

To execute the tests efficiently, tools in the form of test tools are necessary. A test tool is an automated instrument that provides support to one or more test activities, such as planning and control, test specification, and test execution. The use of tools can have the following advantages:

• Increased productivity
• Higher testing quality
• Increased work enjoyment
• Extension of test options.

The test tools are classified in four groups:

• Tools for planning and managing the test
• Tools for designing the test
• Tools for executing the test
• Tools for debugging and analysing the code.

Workplaces

One of the aspects that is often forgotten in testing, is the availability of a workplace where testers can do their job under good conditions, effectively and efficiently. This means office setup in the broadest sense since the testers must also be able to do their work under good conditions. The workplace is therefore more than just office space and a PC. Matters such as access passes, power supply and facilities to have lunch must be arranged. At first sight, the workplace for a tester does not differ much from the regular workplace. But appearances can be deceptive. What is tested is often new to the organisation and the workplace. Testers may have to deal with the situation that their workplace is not yet prepared for the new software. For example, testers often require separate authorisations. They must, for instance, be able to install the new software on their local PC. This may also be necessary for the use of certain test tools.

ORGANISATION

Test processes that are not adequately organised usually have disastrous results. The involvement of many different disciplines (see figure 3), conflicting interests, unpredictability, complex management tasks, lack of experience (figures), and time pressure make setting up and managing the test organisation a complex task. On the one hand there is the organisation in the test team where everyone must have their tasks and responsibilities. On the other, the test team must be an integral part of the project organisation.

A test organisation can be seen as the creation of effective relationships between test functions, test facilities and test activities to issue good quality advice in time.


Figure 3: The many disciplines involved

The organisation of structured testing requires attention for the following fields:

• Test policy
• Permanent test organisation
• Test organisation in projects
• Test professional
• Test roles.

Test policy

The test policy describes how an organisation deals with the people, resources and methods involved in the test process in the various situations. Since testing is one of the tools to ensure quality, the test policy will have to be in line with the other policy measures and initiatives in relation to quality management. We recommend making sure that the test policy is in line with the strategic, tactical and operational policy of the organisation.

Permanent test organisation

A permanent test organisation, contrary to project-based testing, does not elaborate a specific element of the test process on a per project basis, but across all projects. Reasons to create such an organisation are, among other things, the improved leverage of scarce expertise, standardisation of test products, limiting the test project start-up time, continuous improvement of the test process, consolidation of experiences, and prior insight into the test costs and lead time.

Test organisation in projects

At the start of a test project, the roles, tasks, authorisations and responsibilities for the test project are defined. This can be done for the total test process (i.e. across all test levels), or for one specific test level. The relationship between the specified roles, the separate test levels, and the relationships with the other stakeholders in the system development process are then determined and laid down. The test manager must not forget to establish the relationship with a test or quality department, if any.

How this is all set up specifically depends heavily on the organisation type selected for the test work. The choice depends on the test level, project and organisation. The test manager can sometimes – but not always – influence this decision. Roughly, the following organisation types can be distinguished:

• Testing as an autonomous activity in a project
• Testing integrated in a project
• Testing as an autonomous line organisation
• Testing integrated in a line organisation.

Test professional

A great variety of expertise is required for a tester to be able to function well in the discipline of testing. A tester must have knowledge in the field of the subject matter, the infrastructure (test environment, development platform, test tools) and testing itself. What are a tester’s characteristics, in other words, what properties must a person have to be an ideal tester? While many can be listed, the tester must at least:

• have verbal and written communication skills
• be able to work accurately and have analytical skills
• be convincing and persevering
• be factual and have a positively critical attitude
• be creative.

Test roles

The execution of test activities in a project or in the line requires that the tasks are defined and that the executor of the tasks has the right knowledge and competencies. Roles and positions can be distinguished in this respect. A role is the description of one or more tasks with the required knowledge and competencies. There are roles that match positions one-on-one. There are also roles that do not exist as a position.

Differences and similarities between roles and positions:

• A role aims at fulfilling tasks for the test project or permanent test organisation.
• A position focuses on the employee and his place in the career cube.
• They share the tasks to be executed and the required knowledge competencies.